Skip to content

[Store] feat: implements strict multi-tenant quota admission#2612

Merged
ykwd merged 12 commits into
kvcache-ai:mainfrom
Lin-z-w:feat/multi-tenant-init
Jun 30, 2026
Merged

[Store] feat: implements strict multi-tenant quota admission#2612
ykwd merged 12 commits into
kvcache-ai:mainfrom
Lin-z-w:feat/multi-tenant-init

Conversation

@Lin-z-w

@Lin-z-w Lin-z-w commented Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

Part of #2153. Implements strict multi-tenant quota admission for Mooncake Store.

This PR replaces the inherited/default tenant quota model with a connector-backed policy source:

  • Adds enable_multi_tenants, tenant_quota_connector_type, and tenant_quota_connector_uri.
  • Adds TenantQuotaPolicyStore and a v1 YAML file connector with atomic writes.
  • Requires write/allocate requests to use an explicitly registered tenant when multi-tenant mode is enabled.
  • Removes implicit default quota inheritance and the default quota admin endpoint.
  • Persists admin tenant quota changes to the connector before applying them in memory.
  • Restores runtime metadata from snapshots while loading tenant policy from the connector as source of truth.
  • Creates orphan in-memory quota state for metadata tenants missing from policy, allowing reads/removes while blocking new writes.
  • Fixes tenant quota policy update/delete races, YAML round-trip safety, and tenant eviction shard scan fairness.

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Test commands:

cmake --build build --target tenant_quota_test master_service_tenant_quota_test
ctest --test-dir build -R "tenant_quota_test|master_service_tenant_quota_test" --output-on-failure

Test results:

  • Unit tests pass
  • Integration tests pass (if applicable)
  • Manual testing done (describe below)

tenant_quota_test and master_service_tenant_quota_test passed locally.

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit run --all-files and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

Codex was used to help implement, review, test, and document this change. I reviewed and edited the generated code before submission.

@github-actions github-actions Bot added documentation Improvements or additions to documentation run-ci Store labels Jun 25, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Jun 25, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the tenant quota management system to support a strict multi-tenant mode with connector-managed policies (e.g., YAML files), removing default fallback quotas and requiring explicit tenant registration. Key improvements include the introduction of a TenantQuotaPolicyStore for persistent policy management, updated admin APIs, and modified snapshot restore behavior. The review feedback highlights several optimization opportunities in MasterService, including pre-grouping tenant quotas by shard in ApplyTenantQuotaPolicies to reduce complexity, performing the TenantHasObjects check earlier in DeleteTenantQuotaPolicy to simplify rollback logic, and consolidating loops in RecomputeTenantEffectiveQuotas to avoid redundant double-locking of shard mutexes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread mooncake-store/src/master_service.cpp
Comment thread mooncake-store/src/master_service.cpp
Comment thread mooncake-store/src/master_service.cpp
Comment thread mooncake-store/src/master_service.cpp Outdated
@Lin-z-w Lin-z-w force-pushed the feat/multi-tenant-init branch from 6480029 to 1df4fd2 Compare June 25, 2026 05:46
@Lin-z-w Lin-z-w force-pushed the feat/multi-tenant-init branch from e3ad819 to 0bc3b0a Compare June 25, 2026 06:35
@Lin-z-w Lin-z-w marked this pull request as draft June 25, 2026 12:43
@Lin-z-w Lin-z-w marked this pull request as ready for review June 25, 2026 13:07
@codecov-commenter

codecov-commenter commented Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 85.65310% with 201 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
mooncake-store/src/master_service.cpp 77.55% 103 Missing ⚠️
mooncake-store/src/tenant_quota_policy_store.cpp 67.52% 76 Missing ⚠️
...e-store/tests/master_service_tenant_quota_test.cpp 97.24% 12 Missing ⚠️
mooncake-store/include/master_config.h 75.00% 4 Missing ⚠️
mooncake-store/src/master.cpp 60.00% 4 Missing ⚠️
mooncake-store/src/master_admin_service.cpp 66.66% 1 Missing ⚠️
mooncake-store/src/tenant_quota.cpp 93.75% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

yokinoshitayoki
yokinoshitayoki reviewed Jun 26, 2026
View reviewed changes
Comment thread mooncake-store/src/master_service.cpp
Comment thread mooncake-store/src/master_service.cpp Outdated
yokinoshitayoki
yokinoshitayoki reviewed Jun 29, 2026
View reviewed changes
Comment thread mooncake-store/src/master_service.cpp Outdated
yokinoshitayoki
yokinoshitayoki reviewed Jun 29, 2026
View reviewed changes
Comment thread mooncake-store/src/master_service.cpp
yokinoshitayoki
yokinoshitayoki reviewed Jun 29, 2026
View reviewed changes
Comment thread docs/source/design/mooncake-store.md Outdated
yokinoshitayoki
yokinoshitayoki reviewed Jun 29, 2026
View reviewed changes
Comment thread mooncake-store/src/master_service.cpp
yokinoshitayoki
yokinoshitayoki reviewed Jun 29, 2026
View reviewed changes
Comment thread mooncake-store/src/master_service.cpp Outdated
yokinoshitayoki
yokinoshitayoki reviewed Jun 29, 2026
View reviewed changes
Comment thread mooncake-store/tests/master_service_tenant_quota_test.cpp Outdated
yokinoshitayoki
yokinoshitayoki approved these changes Jun 30, 2026
View reviewed changes

@yokinoshitayoki yokinoshitayoki left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@ykwd ykwd merged commit 98ebebf into kvcache-ai:main Jun 30, 2026
23 checks passed
@Lin-z-w Lin-z-w deleted the feat/multi-tenant-init branch June 30, 2026 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yokinoshitayoki yokinoshitayoki yokinoshitayoki approved these changes
@ykwd ykwd Awaiting requested review from ykwd ykwd is a code owner
@stmatengss stmatengss Awaiting requested review from stmatengss stmatengss is a code owner
@XucSh XucSh Awaiting requested review from XucSh XucSh is a code owner
@YiXR YiXR Awaiting requested review from YiXR YiXR is a code owner
@ShangmingCai ShangmingCai Awaiting requested review from ShangmingCai ShangmingCai is a code owner
@Ann-1024 Ann-1024 Awaiting requested review from Ann-1024 Ann-1024 is a code owner
@luketong777 luketong777 Awaiting requested review from luketong777 luketong777 is a code owner
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CI/Build documentation Improvements or additions to documentation run-ci Store

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Lin-z-w @codecov-commenter @yokinoshitayoki @ykwd